Skip to content

feat(wall): support unfiltered suppression and protect crash bootstrap (PROF-12146) - #663

Open
kaahos wants to merge 81 commits into
mainfrom
paul.fournillon/wallclock-all-threads
Open

kaahos wants to merge 81 commits into
mainfrom
paul.fournillon/wallclock-all-threads

Conversation

@kaahos

@kaahos kaahos commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?:

Enables lifecycle-owned wall-clock suppression with wallprecheck=true,filter=. Omitting filter still enables context filtering. Retained native identities let normal-thread hooks identify a blocked run; the timer never registers remote threads.

Also protects crash-handler bootstrap before TLS-based recursion bookkeeping, blocks profiling signals before publishing exit marks, removes TLS work from the wakeup handler, and repairs the bounded GC-generations regression oracle.

Motivation:

An unfiltered recording needs reliable run boundaries before it can omit repeated signals. OS thread state alone cannot prove that a blocked run is unchanged. Uniform reservoir selection therefore precedes suppression; suppressed selections still count toward the epoch denominator, while failed sends do not.

The crash-bootstrap defect is inherited from the guard/TLS lineage in #134, #540 and #713, not caused by unfiltered profiling. A TLS recursion guard cannot protect the first TLS access, including during startup or pthread teardown. Exit marks bypass that access; a Linux fault fence covers the remaining TLS prefix.

Additional Notes:

Human review: mandatory. The change touches signal handlers, the SIGSEGV/SIGBUS crash path, thread teardown and lock-free registries.

Accepted costs (reviewer acknowledgment required):

Cost Where Why accepted
Every SIGSEGV/SIGBUS entry makes a raw identity syscall and scans 64 registry probes; on Linux the fence adds two mask syscalls. This includes JVM-owned faults (implicit null checks, poll pages). Profiler::segvHandler/busHandler A TLS access cannot be made safe before the thread is known to be live. Not yet measured with an NPE-heavy workload.
Sampled native allocations make one raw identity syscall; unsampled ones make none. MallocTracer::maybeRecord shouldSample() reads no TLS, so it runs first. Weights stay unbiased; a sample that lands on the profiler's own allocation is dropped and counted in malloc_samples_dropped.
The owner registry retains up to 2,048 identities with an 8,192-entry TID index; cold registration may allocate or wait on a non-signal mutex. Capacity or timer-probe exhaustion falls back to ordinary sampling. threadFilter.h Bounded memory; failure mode is ordinary sampling, not lost data.
The block tracker's per-slot backing (ThreadFilter::kMaxThreads × sizeof(BlockState), 128 KiB) is retained after stop() and reported in a new NativeMem category, wall_block_tracker. wallClockBlockTracker.cpp, nativeMem.h In-flight owner hooks can carry slot IDs across stop(). wallclock stays recording-scoped and reads 0 when idle, as ProfilerLifecycleTest asserts.
WallClockEpoch.numSuccessfulSamples now counts precheck-suppressed selections, in filtered and unfiltered mode. wallClock.h Keeps the epoch denominator equal to reservoir selections.
  • This is not alternate-stack protection: there is no sigaltstack. Unhooked threads and cancellation paths do not all receive an early exit mark. macOS has no fence.
  • Bundled changes outside the PR's purpose are tracked separately: crash-bootstrap hardening (PROF-16790), OpenJ9/Semeru integration provisioning with musl/OpenJ9 cells reported as not applicable (PROF-16792), _activated_mask in Profiler::stop() and the musl wall-timer guard (PROF-16793, also present on main), and the GCGenerationsTest rewrite (PROF-16794).
  • Merged with current main. main now installs malloc hooks per recording with an identity-checked live-library walk, which supersedes this PR's earlier loader-outside-mutex malloc fix (PROF-16791); that fix and its tests were dropped. main's unowned-tail reset at start and at recording-memory release now delegates to WallClockBlockTracker.
  • Review: Sphinx standard review of the full range found no Critical or High findings; the Low test and doc findings are fixed. The open Low findings are: an exit mark is cleared if an exiting thread re-initializes TLS (no current call site does), and install-openj9.sh leaves set -euo pipefail active in the sourcing job. The last-probe registry tests hardcode the 4096-slot table and 64-probe window instead of shared constants. The wall_block_tracker no-growth check in ProfilerLifecycleTest only runs in a CPU-only restart test, and WallClockAccountingTest checks a fixture-local tracker, so both guard the category only weakly.

How to test the change?:

  • ./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug (crash bootstrap, thread filter, wall clock, block tracker, malloc admission).
  • ./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests='com.datadoghq.profiler.wallclock.*' and -Ptests='com.datadoghq.profiler.nativemem.*' on glibc and musl.
  • bash .gitlab/dd-trace-integration/tests/post_pr_comment_test.sh, bash .gitlab/dd-trace-integration/tests/validate_quarantine_test.sh, python3 .gitlab/dd-trace-integration/tests/test_jvm_selection.py.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-12146

Unsure? Have a question? Request a review!

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #44814402397 | Commit: e498d8a | Duration: 22m 28s (longest job)

✅ All 108 test jobs passed

Status Overview

JDK glibc-aarch64/asan glibc-aarch64/debug glibc-aarch64/release glibc-aarch64/tsan glibc-amd64/asan glibc-amd64/debug glibc-amd64/release glibc-amd64/tsan musl-aarch64/debug musl-aarch64/release musl-amd64/debug musl-amd64/release
8 - - - - ✅ ✅ ✅ ✅ - - - -
8-ibm - - - - ✅ ✅ ✅ ✅ - - - -
8-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
8-librca - - - - - - - - ✅ ✅ ✅ ✅
8-orcl - - - - ✅ ✅ ✅ ✅ - - - -
11 - - - - ✅ ✅ ✅ ✅ - - - -
11-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
11-librca - - - - - - - - ✅ ✅ ✅ ✅
17 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-j9 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
17-librca - - - - - - - - ✅ ✅ ✅ ✅
21 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
21-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
21-librca - - - - - - - - ✅ ✅ ✅ ✅
25 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
25-graal ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ - - - -
25-librca - - - - - - - - ✅ ✅ ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 108 | Passed: 108 | Failed: 0


Updated: 2026-11-20 21:55:08 UTC

@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch 3 times, most recently from 3902a14 to f302dd5 Compare July 19, 2026 12:00
@kaahos kaahos changed the title feat(wall): add configurable wall thread scope feat(wall): support signal suppression for unfiltered threads Jul 19, 2026
@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch 2 times, most recently from 0ad425c to fb1c44d Compare July 19, 2026 18:43
@dd-octo-sts

dd-octo-sts Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit fb1c44d)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125547869 Commit: fb1c44de77197780eae60267ff98bd38a907a558

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +5.9% (2679→2836 ms)
  • 🔴 future-genetic (JDK 21): runtime +3.8% (2083→2162 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10210 ms (21 iters) ✅ 10256 ms (21 iters) ≈ +0.5% (±10.9%) — / —
akka-uct 25 ✅ 8953 ms (24 iters) ✅ 8890 ms (24 iters) ≈ -0.7% (±10.2%) — / —
finagle-chirper 21 ✅ 6000 ms (33 iters) ✅ 5960 ms (33 iters) ≈ -0.7% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5431 ms (36 iters) ✅ 5443 ms (36 iters) ≈ +0.2% (±24.6%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2679 ms (70 iters) ✅ 2836 ms (66 iters) 🔴 +5.9% — / —
fj-kmeans 25 ✅ 2847 ms (66 iters) ✅ 2828 ms (66 iters) ≈ -0.7% (±2.6%) — / —
future-genetic 21 ✅ 2083 ms (89 iters) ✅ 2162 ms (86 iters) 🔴 +3.8% — / —
future-genetic 25 ✅ 2076 ms (90 iters) ✅ 2122 ms (88 iters) ≈ +2.2% (±2.5%) — / —
naive-bayes 21 ✅ 1252 ms (136 iters) ✅ 1248 ms (137 iters) ≈ -0.3% (±32.7%) — / —
naive-bayes 25 ✅ 1008 ms (170 iters) ✅ 1023 ms (167 iters) ≈ +1.5% (±31.7%) — / —
reactors 21 ✅ 16312 ms (15 iters) ✅ 16357 ms (15 iters) ≈ +0.3% (±8.8%) — / —
reactors 25 ✅ 18416 ms (15 iters) ✅ 18780 ms (15 iters) ≈ +2% (±3.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 1891 / 2004 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2305 / 2283 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 2 8183 / 8674 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 8087 / 8641 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1270 / 1278 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 3 1282 / 1275 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 3 2966 / 3007 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2992 / 2972 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 6 3491 / 3501 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 5 3488 / 3493 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 1711 / 1786 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1819 / 1920 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from fb1c44d to 1118f62 Compare July 19, 2026 21:16
@dd-octo-sts

dd-octo-sts Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 1118f62)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125554643 Commit: 1118f62831a4384bdf0283ef01a626c5bdc48af5

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -3.9% (2808→2698 ms)
  • 🟢 future-genetic (JDK 25): runtime -4.2% (2088→2001 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10294 ms (21 iters) ✅ 10251 ms (21 iters) ≈ -0.4% (±11.4%) — / —
akka-uct 25 ✅ 8933 ms (24 iters) ✅ 8835 ms (24 iters) ≈ -1.1% (±10%) — / —
finagle-chirper 21 ✅ 6016 ms (33 iters) ✅ 6034 ms (33 iters) ≈ +0.3% (±25.3%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5462 ms (36 iters) ✅ 5452 ms (36 iters) ≈ -0.2% (±24.6%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2808 ms (66 iters) ✅ 2698 ms (70 iters) 🟢 -3.9% — / —
fj-kmeans 25 ✅ 2812 ms (66 iters) ✅ 2813 ms (66 iters) ≈ +0% (±2.6%) — / —
future-genetic 21 ✅ 2067 ms (90 iters) ✅ 2083 ms (90 iters) ≈ +0.8% (±2.7%) — / —
future-genetic 25 ✅ 2088 ms (89 iters) ✅ 2001 ms (93 iters) 🟢 -4.2% — / —
naive-bayes 21 ✅ 1239 ms (138 iters) ✅ 1231 ms (138 iters) ≈ -0.6% (±32.7%) — / —
naive-bayes 25 ✅ 1019 ms (168 iters) ✅ 1011 ms (169 iters) ≈ -0.8% (±32.1%) — / —
reactors 21 ✅ 16149 ms (15 iters) ✅ 16215 ms (15 iters) ≈ +0.4% (±8.4%) — / —
reactors 25 ✅ 18598 ms (15 iters) ✅ 18345 ms (15 iters) ≈ -1.4% (±5.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 2 1978 / 1973 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2287 / 2449 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8978 / 8724 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 8633 / 8297 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 1 1257 / 1281 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 4 1280 / 1268 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 3 2909 / 2923 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 5 2847 / 2887 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 7 / 4 3492 / 3493 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 2 3463 / 3473 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1761 / 1721 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1916 / 1880 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 40dbe8e)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125596529 Commit: 40dbe8e833888192b6026558590bce1cf0ad1c9f

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -2.7% (2823→2748 ms)
  • 🔴 future-genetic (JDK 21): runtime +4.1% (2041→2125 ms)
  • 🔴 future-genetic (JDK 25): runtime +5.8% (1994→2110 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10340 ms (21 iters) ✅ 10312 ms (21 iters) ≈ -0.3% (±11.6%) — / —
akka-uct 25 ✅ 8871 ms (24 iters) ✅ 8849 ms (24 iters) ≈ -0.2% (±10.1%) — / —
finagle-chirper 21 ✅ 6017 ms (33 iters) ✅ 6015 ms (33 iters) ≈ -0% (±25%) ⚠️ W:3 / ⚠️ W:4
finagle-chirper 25 ✅ 5470 ms (36 iters) ✅ 5509 ms (36 iters) ≈ +0.7% (±24.2%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2823 ms (66 iters) ✅ 2748 ms (68 iters) 🟢 -2.7% — / —
fj-kmeans 25 ✅ 2856 ms (66 iters) ✅ 2847 ms (66 iters) ≈ -0.3% (±2.6%) — / —
future-genetic 21 ✅ 2041 ms (90 iters) ✅ 2125 ms (88 iters) 🔴 +4.1% — / —
future-genetic 25 ✅ 1994 ms (93 iters) ✅ 2110 ms (88 iters) 🔴 +5.8% — / —
naive-bayes 21 ✅ 1247 ms (137 iters) ✅ 1259 ms (135 iters) ≈ +1% (±33.5%) — / —
naive-bayes 25 ✅ 1027 ms (167 iters) ✅ 1018 ms (168 iters) ≈ -0.9% (±31.5%) — / —
reactors 21 ✅ 16583 ms (15 iters) ✅ 16798 ms (15 iters) ≈ +1.3% (±7.5%) — / —
reactors 25 ✅ 18582 ms (15 iters) ✅ 18610 ms (15 iters) ≈ +0.2% (±5.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / 6 1978 / 1998 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2344 / 2271 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8863 / 8312 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 8368 / 8738 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 7 / 1 1284 / 1242 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 4 1302 / 1289 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 1 2979 / 2980 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 2847 / 2862 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 5 3520 / 3534 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 3 / 3 3523 / 3509 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 2 1682 / 1809 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1806 / 1871 ✅ / ✅ ✅ / ✅

Copilot AI review requested due to automatic review settings July 21, 2026 07:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Extends wall-clock “owned-block” signal suppression to unfiltered (explicit filter=) wall recordings when wallprecheck=true, by introducing a thread registry keyed by native TID and integrating it into both ASGCT and JVMTI wall-clock sampling paths.

Changes:

  • Added unfiltered wall thread registry support (native TID indexing, lifecycle generations, recording epochs) and wired it into profiler lifecycle + wall-clock precheck logic.
  • Added a bounded candidate-selection/backfill mechanism for precheck to avoid reverting to O(N) registry work per wall tick under heavy suppression.
  • Added Java integration tests, C++ unit tests, and a JMH benchmark to validate behavior and measure overhead.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java New integration coverage for suppression behavior in explicit-empty-filter wall recordings.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java Verifies registry activation does not leak across recording restarts and handles stopped gaps safely.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java Ensures the unfiltered path works when wall sampling delegates stacks via JVMTI.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java Confirms J9 wall engine capability gating prevents activation of unfiltered tracking.
ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java Adds beforeProfilerStart() hook to allow test setup before profiler thread callbacks are enabled.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java Exposes owned-block hooks to benchmarks without widening core API surface.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java Adds throughput benchmark comparing wallprecheck=false/true as owned-block population grows.
ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp Adds capability tests and verifies explicit-empty vs omitted filter parsing behavior.
ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp Unit-tests the new bounded candidate selection logic.
ddprof-lib/src/test/cpp/threadFilter_ut.cpp Expands tests for registry/epoch/lifecycle behavior and adjusts TID usage in recovery test.
ddprof-lib/src/main/cpp/wallClockCandidateSelector.h Adds reusable “randomized prefix without replacement” candidate selection with stats.
ddprof-lib/src/main/cpp/wallClock.h Integrates candidate selector and adds bounded-visit backfill path to the wall timer loop.
ddprof-lib/src/main/cpp/wallClock.cpp Updates precheck suppression to validate registry identity + context transitions; supports lazy registry lookups.
ddprof-lib/src/main/cpp/threadFilter.h Introduces registry state (TID index, epochs, lifecycle generation) and APIs for unfiltered tracking.
ddprof-lib/src/main/cpp/threadFilter.cpp Implements lock-free lookups + mutex-serialized writers for TID indexing, epoch refresh, and retirement.
ddprof-lib/src/main/cpp/profiler.h Declares bootstrap of existing Java threads for registry population.
ddprof-lib/src/main/cpp/profiler.cpp Boots existing Java threads into registry when unfiltered tracking is active; deactivates registry on stop/failure.
ddprof-lib/src/main/cpp/jvmThread.h Adds capability check for cross-thread native TID lookup support.
ddprof-lib/src/main/cpp/jvmThread.cpp Implements native TID lookup support predicate.
ddprof-lib/src/main/cpp/javaApi.cpp Ensures TLS slot binding is validated/refreshed before using owned-block and filter APIs.
ddprof-lib/src/main/cpp/engine.h Adds wall-engine capability hook for unfiltered precheck support.
ddprof-lib/src/main/cpp/counters.h Adds counters for registry bootstrap and bounded-lookup/backfill diagnostics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit d82291f)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125875202 Commit: d82291ff62852b63091350a05ebf26283fa39029

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10141 ms (21 iters) ✅ 10278 ms (21 iters) ≈ +1.4% (±11.9%) — / —
akka-uct 25 ✅ 8800 ms (24 iters) ✅ 8939 ms (24 iters) ≈ +1.6% (±9.7%) — / —
finagle-chirper 21 ✅ 6003 ms (33 iters) ✅ 6060 ms (33 iters) ≈ +0.9% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5463 ms (36 iters) ✅ 5482 ms (36 iters) ≈ +0.3% (±24.1%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2780 ms (68 iters) ✅ 2771 ms (68 iters) ≈ -0.3% (±2.7%) — / —
fj-kmeans 25 ✅ 2752 ms (68 iters) ✅ 2824 ms (66 iters) ≈ +2.6% (±2.7%) — / —
future-genetic 21 ✅ 2082 ms (90 iters) ✅ 2035 ms (91 iters) ≈ -2.3% (±2.6%) — / —
future-genetic 25 ✅ 2101 ms (88 iters) ✅ 2055 ms (90 iters) ≈ -2.2% (±2.6%) — / —
naive-bayes 21 ✅ 1299 ms (132 iters) ✅ 1232 ms (138 iters) ≈ -5.2% (±31.7%) — / —
naive-bayes 25 ✅ 1011 ms (169 iters) ✅ 1013 ms (169 iters) ≈ +0.2% (±31.4%) — / —
reactors 21 ✅ 16511 ms (15 iters) ✅ 16445 ms (15 iters) ≈ -0.4% (±9.8%) — / —
reactors 25 ✅ 18516 ms (15 iters) ✅ 18146 ms (15 iters) ≈ -2% (±6.2%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 2 1925 / 1914 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 3 2117 / 2331 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 3 8401 / 8842 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8557 / 8073 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 3 1243 / 1272 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1265 / 1283 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / 1 3054 / 3036 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2890 / 2867 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 3 3522 / 3464 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 5 3452 / 3463 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 2 1679 / 1552 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1933 / 1805 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit ec76f58)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125883187 Commit: ec76f5829d243f516dee70833f3dfbe7a2b9eaa3

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -4.1% (2839→2724 ms)
  • 🔴 future-genetic (JDK 21): runtime +3% (2066→2127 ms)
  • 🔴 future-genetic (JDK 25): runtime +5.8% (2006→2122 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10350 ms (21 iters) ✅ 10305 ms (21 iters) ≈ -0.4% (±11.3%) — / —
akka-uct 25 ✅ 8864 ms (24 iters) ✅ 8976 ms (24 iters) ≈ +1.3% (±10.4%) — / —
finagle-chirper 25 ✅ 5472 ms (36 iters) ✅ 5418 ms (36 iters) ≈ -1% (±24.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2839 ms (66 iters) ✅ 2724 ms (68 iters) 🟢 -4.1% — / —
fj-kmeans 25 ✅ 2851 ms (66 iters) ✅ 2836 ms (66 iters) ≈ -0.5% (±2.6%) — / —
future-genetic 21 ✅ 2066 ms (89 iters) ✅ 2127 ms (87 iters) 🔴 +3% — / —
future-genetic 25 ✅ 2006 ms (92 iters) ✅ 2122 ms (87 iters) 🔴 +5.8% — / —
naive-bayes 21 ✅ 1277 ms (134 iters) ✅ 1289 ms (133 iters) ≈ +0.9% (±32.4%) — / —
naive-bayes 25 ✅ 1021 ms (167 iters) ✅ 1023 ms (168 iters) ≈ +0.2% (±31.6%) — / —
reactors 21 ✅ 15903 ms (15 iters) ✅ 16452 ms (15 iters) ≈ +3.5% (±8.8%) — / —
reactors 25 ✅ 18032 ms (15 iters) ✅ 18628 ms (15 iters) ≈ +3.3% (±4.7%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 3 2089 / 2428 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 7 / 2 8520 / 8487 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 4 / 2 8382 / 8319 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 4 / 4 1278 / 1277 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 4 / ✅ 1290 / 1289 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / 2 2953 / 3032 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 2 2846 / 2856 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 3 3534 / 3561 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 3 3480 / 3510 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 1614 / 1718 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 3 1962 / 1876 ✅ / ✅ ✅ / ✅

@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
@DataDog DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026
kaahos added a commit that referenced this pull request Nov 18, 2026
Do not schedule musl/OpenJ9 integration jobs: upstream publishes no musl runtime. Report these combinations as N/A while retaining failure and completeness checks for applicable configurations.

Applies to #663.
kaahos added a commit that referenced this pull request Nov 18, 2026
Do not schedule musl/OpenJ9 integration jobs: upstream publishes no musl runtime. Report these combinations as N/A while retaining failure and completeness checks for applicable configurations.

Applies to #663.
kaahos added a commit that referenced this pull request Nov 18, 2026
Do not schedule musl/OpenJ9 integration jobs: upstream publishes no musl runtime. Report these combinations as N/A while retaining failure and completeness checks for applicable configurations.

Applies to #663.
kaahos added a commit that referenced this pull request Nov 18, 2026
Do not schedule musl/OpenJ9 integration jobs: upstream publishes no musl runtime. Report these combinations as N/A while retaining failure and completeness checks for applicable configurations.

Applies to #663.
kaahos added a commit that referenced this pull request Nov 18, 2026
Do not schedule musl/OpenJ9 integration jobs: upstream publishes no musl runtime. Report these combinations as N/A while retaining failure and completeness checks for applicable configurations.

Applies to #663.
kaahos added a commit that referenced this pull request Nov 18, 2026
Do not schedule musl/OpenJ9 integration jobs: upstream publishes no musl runtime. Report these combinations as N/A while retaining failure and completeness checks for applicable configurations.

Applies to #663.

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR adds unfiltered wall-sampling suppression; ELF-header validation, teardown hook admission, signal ordering, class-loader access, and CI reporting are fixed. One issue remains: glibc/OpenJ9 provisioning still uses malformed Semeru release URLs, so those integration jobs fail before testing.

Open Bits AI session

🤖 Bits Code Review · Commit bd58934

@kaahos

kaahos commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

In response to the attributionPC report: 041b24a0e uses uintptr_t subtraction and covers null and 0x1 return addresses. The current #663 head contains that implementation, not the obsolete null-only guard.

@kaahos kaahos added the human_review:mandatory Human review required — see PR description label Nov 19, 2026
@kaahos kaahos changed the title feat(wall): support unfiltered suppression and protect crash bootstrap (PROF-16669) feat(wall): support unfiltered suppression and protect crash bootstrap (PROF-12146) Nov 19, 2026

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR adds unfiltered wall-sampling suppression and teardown protection; earlier class-loader, fault-recovery, ELF-validation and CI-reporting concerns are resolved. Two issues remain: malformed Semeru URLs prevent OpenJ9 integration provisioning, and the GC-generations restart test expects liveness samples that stop has already discarded.

Open Bits AI session

🤖 Bits Code Review · Commit a2c4a46

matches[0]++;
}
})[0];
assertTrue(historical > 0, "retained entries from the previous recording are present");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Do not expect liveness entries to survive stop/start

On supported HotSpot configurations, stopProfiler() reaches Profiler::releaseRecordingMemory(), whose LivenessTracker::release() deletes the weak references and frees the tracking table. setupProfiler() then creates a fresh recording with empty tracking state; retaining the old arrays does not sample their allocations again. Consequently historical is zero and this assertion fails. Exercise historical filtering without assuming liveness entries survive a recording stop, or use controlled event fixtures.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

human_review:mandatory Human review required — see PR description sphinx:critical Sphinx: critical — human review required test:asan Run CI tests with AddressSanitizer configuration test:fuzz test:release Run CI tests with release configuration test:reliability test:tsan Run CI tests with ThreadSanitizer configuration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants